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Motivation for Music Visualization 


Motivation 


The motivation for this project is to create a visualization that accurately 
represents multiple components of musical pieces. 


Generally speaking, this project enables the appreciation of a concept, in 
this case music, through more than one human sense. 


Additionally, parts of this project can be used and modified to develop new 
software, such as a music sheet creator. 


Theory 
Theory for the Music Visualization project 


Theory 


Source Separation: 


A classical example for source separation is the "cocktail party problem", 
where a number of people talk simultaneously and one is trying to follow 
one of the discussions. 


In our case, we deal with music instead of multiple people. Music is often 
composed of multiple instruments playing different notes at the same time. 
In order to separate these instruments, there exist many methods in signal 
processing which perform source separation, attempting to determine the 
different signals that make up the music. 


Some of the most famous methods of source separation include 
Independent Component Analysis (ICA), Markov chains, and using Fast 
Fourier Transforms (FFTs). For this project we decided to focus on ICA 
instead of other methods to separate the sources, since it could be 
implemented in Matlab easily. 


ICA Algorithm: 


Independent Component Analysis (ICA) takes a 0.5 second frame of the 
track, sliding this 0.5 second window along the entire song. It then separates 
this frame into 20 possible sources, determines 3 sources that are maximally 
independent, and returns those 3 sources. We then perform FFTs on these 3 
sources and find the peaks from the resulting spectrums of the signals. 
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Fisqiency (Hz) 


From these frequencies, we determine the peaks, take the two most dominat 
peaks per source, and create matrix containing the results from each frame. 


Relevant Frequency Values After ICA 
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If we see a note that lasts for more than 5 frames, we call it a note and 
visualize it. In the example above, we can see a constant note in low 
frequencies, this could be considered the bass line. 


Our ICA algorithm is based off of JadeR algorithm, which was 
implemented in our Matlab code. 


There are many factors we can change in our Matlab code, including: 


¢ Number of sources, though this lowers the height of peaks when we 
find the FFT. 

e Threshold for what we consider "peaks" in our FFTs 

e¢ Number of frames necessary for a frequency to be considered a note. 


Design of Visualization 
Design of the Music Visualization Project 


Design of Visualization 


General Design: 


Our analysis of the music was performed in Matlab, whilst our 
visualizations were run using Java, with an IDE called Processing, which 
contains a user friendly graphics library. 


A major part of our design was choosing how to map the data from our 
results in Matlab to our visualizations in Java. Once the mapping was 
chosen, we had to keep our visualizations synced with the music, and 
maintain a constant framerate in the visualizations. An example of our 
visualization without any input is shown below: 


Using Matlab: 


We chose to get the two most dominant components from every source, 
giving us a total of six components per frame, and chaining them together 
to determine what our "notes" were. Then, we displayed them as falling 
rain, or lines, from the top of the screen, with their frequency determining 
where they appear (low frequencies to the left, high frequencies to the 
right), and their duration determining the length of the line that appears. 


Using Java: 


We chose to use the FFT of the song to control the color wheel at the center 
of the screen. The low frequencies of the song were mapped closer to blue, 
whilst the higher frequencies were closer to red. The amplitude of each 
frequency determines the radial expansion of each line in our color wheel. 


Finally, we used a process called BeatDetect, which was part of the library 
in Processing, to create our last animation. BeatDetect ran FFT on two 
(normally one) low frequency bands of the song to determine the higher 
amplitudes found, and calls them a beat, then, at each beat in the song it 
will release around 200 particles radially from the center of the color wheel. 


Conclusion 
Results, shortcomings, and future work for our Music Visualization project. 


Conclusion 


Results 


Using ICA algorithms, Matlab, Java, and a graphic library, we were able to 
create a our own visualization of music. 


The ICA algorithm in our Matlab code successfully extracted three tracks 
form the song that were independent enough to represent the notes from the 
song. 


The FFT of the song gave us relevant frequencies which were mapped to 
the visualization. 


Our beat detection worked well and shot out particles whenever there were 
beats. 
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Shortcomings 


Our music visualization worked well with Drum and Bass music and 
Instrumental music. However, the ICA algorithm couldn't handle slightly 
wavering notes, which made voice and vibrato troublesome to work with. 


At the same time, the our algorithm couldn't determine harmonics in an 
instrument, and thus would occassionally display the same note being in 
two different places. 


The algorithm used the relative volume of what was going on in each 
frame, and thus would occassionally display sustained notes as a dashed 
line in our code instead of a continuous one. 


Future Developments 


There are many developments that can be implemented to this project, from 
more visualizations for more parts of the song to using different algorithms 
to get more accurate sources. We can also include that smoothing out the 
detection of notes on our current algorithm would benefit the display on our 
visualization, so our lines appear more continuous when it is the same note. 


The most important part that we wish to implement is to have all of our 
code in Java. 


This would allow us to perform the visualization in real time, instead of 
preprocessed data from Matlab. 


Adding an interface to our visualization, with simple things such as 
scrolling through the song, to changing the amount of sources without 
having to go to the code and changing it. 


Another development that could be added is doing voice visualization, 
which at the moment is not easy to do, since the voice usually wavers 
slightly and thus our algorithm doesn't find it as a continuous note. 


Future work for this project includes adding an interface for our visualizer. 
In said interface we could change. 


Code for Visualization 
Code for our Music Visualizer 


Java Processing Code for Visualization 
In the zip file, the code that was used to create the visuals for the Music 
Visualization project 


